PLURAL-MODE SURVEILLANCE SYSTEM AND METHODOLOGY WITH 
DIFFERENTL^TED, SELECTABLE, TWIN-OUTPUT DISPLAY 
Cross-Reference to Related Application 
This application claims priority to U.S. Provisional Patent Application Serial No. 
5 60 /484,264 , filed June 30, 2003, for "Surveillance Imaging System and Methodology". 
The entirety of this priority patent application is hereby incorporated herein by reference. 

Background and Summarv of the Invention 
This invention pertains to a plural-mode, multi-information-character surveillance 
imaging system and methodology. In particular, it relates to such a system and 
10 methodology which feature the use of a differentiated, selectable, twin-visual-output 
display arrangement, and which enable special comparative viewing, via this 
arrangement, of surveillance imagery collected by different ones of several, different- 
mode imagers. For the purpose of illustration herein, a preferred and best mode 
embodiment of, and maimer of practicing, the invention are described in the setting of an 
15 overall surveillance imaging system which employs nighttime, daytime and thermal 
surveillance imaging structures and modalities. The preferred implementation of the 
system also features a very simple, one-hand-operable, computer-based controller which 
offers touch-screen and joystick functionahties that enable very sophisticated, quick and 
accurate user "manipulation" of system behavior in ways that allow virtually 
20 unchallenged operator visual attention to be paid to the twin-output, mode-comparative 
visual displays proposed by the invention. 

In the practice of surveillance imagery utilizing different modes of imagery, such 
as the three modes mentioned above, it is very useful under many different kinds of 
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circumstances to be able to make a simultaneous, or near simultaneous, visual 
comparison between imagery data derived from two different modes. For example, 
during daylight hours, it may be important to have both color optical and thermal imagery 
available respecting the same scene in order to obtain better information about what may 

5 be pictured in that scene that might be of some surveillance concern. Similarly, at 
nighttime, a like kind of comparison might be desired between light-intensified nighttime 
imagery and thermal imagery. These are just two of many other relevant illustrations. 

Proposed structurally by the present invention is a surveillance imaging system 
which very easily accommodates these considerations by furnishing, in an operational 

10 setting where imagery is available at different times selectively from a thermal imager, 
from a daytime, color video imager, and from a light-intensified, black-and-white, 
nighttime imager, a pair of closely spaced side-by-side display devices on whose screens 
different kinds of comparative imagery derived from these imagers can be presented. 

Another illustrative situation wherein side-by-side comparative surveillance 

15 imagery may be important involves information that is desirable to obtain under certain 
kinds of challenging light-of-day conditions which typically exist around daybreak and 
twilight. Here there are circumstances where it would be very useful to be able to view, 
in addition to thermal imagery, a comparative cross-relationship between daytime color 
imagery and nighttime light-intensified imagery. As will be seen, the system and 

20 methodology of this invention readily accommodate this challenging circumstance. 

The various features and other advantages that are offered by the system and 
methodology of this invention will become more fully apparent as the description which 
now follows is read in conjunction with the accompanying drawings. 
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Description of the Drawings 
Fig. 1 A is a simplified and stylized isometric view of a multi-imager, plural-mode 
surveillance system which employs a pair of side-by-side imaging display devices which 
are constructed, and employed methodologically, in accordance with the present 
5 invention. At the right side of this figure, fragmentary dash-double-dot lines illustrate 
one modified form of the system shown centrally in the figure. 

Fig. IB is a simplified block/schematic illustration of another modified form of 
the system centrally pictured in Fig. 1 A. 

Fig. 2 is a view of portions of the system illustrated in Fig. 1 A focusing attention 
10 on selectable, dual side-by-side display structures which operate in accordance with 
preferred practice of the present invention. 

Fig. 3 pictures a computer-generated display on a user-interface touch screen in a 
computer-based controller which is employed in the system of Fig. lA. This figure 
shows a typical touch-screen appearance for a situation where currently co-active in the 
15 system of, this invention are a daytime color imager and a thermal imager whose 
respective outputs are being fed simultaneously to a pair of side-by-side visual display 
devices provided in accordance with this invention. 

Fig. 4 is similar to Fig. 3, except that here what is shown is a typical touch-screen- 
display provided in the system of Fig. lA where currently coactive in the imaging system 
20 shown in Fig. 1 A are a light-intensified, black-and-white, nighttime imager, and a 
thermal imager. 
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Fig. 5 is similar to Figs. 3 and 4, except that here what is shown is a user-interface 
touch screen presentation active under circumstances where only a thermal imager in the 
system of Fig. lA is active. 

Figs. 6 and 7 illustrate same-scene, comparative, thermal and daytime imagery, 
5 respectively, of a helicopter in flight. 

Figs. 8 and 9 illustrate same-scene, comparative, nighttime light-intensified and 
thermal imagery, respectively, of a walking person 

Figs. 10, 11 and 12 illustrate same-scene, comparative, thermal, daytime, and 
nighttime, hght-intensified imagery, respectively, viewing a helicopter in flight during 
10 twiUght. Fig. 10 shows a dedicated thermal image portrayed on the screen in one visual 
display device provided according to the invention. Figs. 11 and 12 picture selectively 
altemate "daytime" and "nighttime-intensified" images portrayed on the screen in a side- 
by-side adjacent, second visual display device provided in accordance with the invention. 

15 Detailed Description of the Invention 

Turning attention now to the drawings, and referring first of all to Fig. lA, 
indicated generally at 10 is a surveillance imaging system which includes twin-output 
display capability in accordance with a preferred and best mode embodiment of, and 
manner of practicing, the present invention. System 10 is referred to herein as a multi- 

20 information-character surveillance imaging system. Included in system 10 are a housing 
structure, or housing, 12 which is appropriately environmentally sealed, and which 
contains a plural-imager assembly of surveillance imagers including (a) a nighttime, 
Ught-intensified, black-and-white imager 14, (b) a thermal imager 16, and (c) a daytime, 
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color video camera imager 18. While the structure of nighttime imager 14 per se forms 
no special part of the present invention, it is worth noting that this imager is itself a 
unique structure which employs the combination of an otherwise conventional light 
intensifier that feeds intensified night scene imagery to the single CCD optical-to- 

5 electronic device contained in an otherwise conventional black-and-white video camera. 
This camera then produces a rich, black-and-white, intensified nighttime image which is 
presented, when selected, on a display device (soon to be mentioned) contained in system 
10 as an otherwise conventional looking black-and-white image. This black-and-white 
image lacks the usual harshness and fatiguing characteristics of a conventional, green- 

10 spectrum, light-intensified image. The process performed in this imager can be described 
as gathering and intensifying night scene data to generate a green-spectrum derivative, 
and converting that derivative to a black-and-white video signal sub-derivative. 

Drivingly connected to housing 12, which housing is suitably supported on a 
stand (not shown), are two computer-controllable electrical motors 20, 22. Motor 20 is 

15 selectively operable by an operator/user of system 10 to cause housing 12 (and the 
contained assembly of imagers) to swing as a unit reversibly back-and-forth angularly (in 
yaw motion) about a generally upright axis shown at 12a. Such swinging motion is 
generally indicated in Fig. lA by double-ended, curved arrow 24 in this figure. 
Similarly, motor 22 is likewise selectively operable to cause reversible up-and-down 

20 angular tilting (a pitch motion) of housing 12, and of the contained imagers, about a 
generally horizontal axis 12b. This motion is indicated by double-ended, curved arrow 
26 in Fig. lA. Suitably interposed housing 12 and the mentioned (but not illustrated) 
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stand, is conventional motion/articulating structure (also not shown) which enablingly 

supports housing 12 on the stand for such motions. 

Each of imagers 14, 16, 18 is provided with suitable computer-controllable 

apparatus for effecting selectable changes in various parameters, such as magnification, 
5 field of view, focus, and any other appropriate operational parameters. The exact 

parameters which are associated controllably with each of imagers 14, 16, 18 do not form 

any part of the present invention. 

Further describing generally the assembly of the three imagers, imagers 14, 16, 18 

are commonly bore-sighted, or bore-sight aligned, along their respective optical axes 14a, 
10 16a, 18a, at infinity which is represented schematically at 19 on the left side of Fig. lA. 

The terminology "commonly bore-sighted" refers to the fact that, effectively at infinity, 

all three imagers are aimed substantially exactly at the same point in space. This special 

co-alignment assures, importantly, a substantially commonly shared point of view for all 

of the imagers. The existence of this shared-point-of-view characteristic, when coupled 
15 in system 10 with side-by-side comparative image-viewing capability, as will be 

explained, greatly enhances the surveillance infonnation-giving capabilities of the 

system. 

Further included in system 10 are (a) a user-operable controller 28 having a 
touch-sensitive screen 28a, and a multi-axis, manual, mechanical joystick shown at 28b, 
20 (b) an appropriate computer 30, (c) video signal switching structure 32, and (d), in 
accordance with the present invention, a pair of side-by-side video screen display devices 
34, 36, also referred to herein as a twin-output display, and as adjacent, co-viewable 
display structure. Device 34 is also called herein a dedicated display structure, and 


6 


OAER.1019 


device 36 a changeably dedicatable device. It is through controller 28 that a 
user/operator of system 10 exercises selective control over just what comparative 
surveillance imagery is presented, as will be seen, by devices 34, 36.. 

Within controller 28, touch screen 28a, through appropriate programming which 

5 is managed by computer 30, which computer is appropriately operatively coupled (not 
specifically shown) to controller 28, enables a user to select and control, among other 
things, the various operating parameters of imagers 14, 16, 18. Such control includes, for 
example, switching these imagers into and out of operation, adjusting focus, establishing 
magnification and thus field of view, and making changes in any other appropriate 

10 parameters. Manual joystick 28b is rockable in manners generally indicated by double- 
ended, curved arrows 28c, 28d to effect housing pitch and yaw angular motions, 
respectively, of the housing and imager assembly via motors 22, 20, respectively. While 
a manual joystick is specifically shown in controller 28, it should be understood that joy- 
stick functionality may, if desired, be provided in a virtual sense by way of an appropriate 

15 touchable screen image provided on touch screen 28a under the control of computer 30. 

Appropriately associated computer-active control lines 38, 40, 42, 44 extend 
operatively as shown between housing 12 (and the imagers contained therein), motors 20, 
22, controller 28, computer 30, and switching structure 32. It is through these lines that 
control is exercised, via controller 28 and the operation of computer 30, over the imagers' 

20 parameter adjustments, the motor operations, and the operations of switching structure 32 
which latter "operations" determine which particular comparative modes of imagery are 
presented at any given time by devices 34, 36. Three additional lines 46, 48, 40 are 
shown extending between housing 12 and switching structure 32, and another line 52 is 
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shown interconnecting structure 32 and display device 36. Still another line 54 is shown 
interconnecting housing 12 and display device 34. Controller 28, computer 30, switching 
structure 32, and the various interconnecting "lines" which are pictured in Fig. lA, 
collectively constitute interconnect structure herein. 
5 Focusing attention for a moment on Fig. 2, here one sees the fundamental 

operating relationship which exists between (a) imagers 14, 16, 18, (b) switching 
structure 32, (c) side-by-side display devices 34, 36, and (d) interconnecting and 
associated lines 44, 46, 48, 50, 52, 54. Computer-controlled switching operation of 
structure 32 (via line 44) is here pictured clearly and schematically. The dedicated 
10 thermal-imagery condition of display device 34 is plainly evident, as is also the 
changeable dedicateability of device 36 to the output imagery data stream of any one of 
the three imagers. 

Lines 46, 48, 50 carry video output signals (data streams) from imagers 14, 16, 
18, respectively, to switching structure 32. Under the control of touch screen 28a and 

15 computer 30, a user/operator can selectively send a signal from any one of these three 
imagers over line 52 for display of an image on display device 36. Thus display device 
36 can selectively and changeably display an image either from nighttime imager 14, 
from thermal imager 16, or from daytime imager 18. Line 54 dedicatedly delivers video 
output image information from thermal imager 16 directly to video display device 34. 

20 As has been noted above, there are many surveillance applications wherein it is 

especially useful and important to have available two, side-by-side display devices 
incorporated into a system, like system 10. Significantly, with this arrangement, as will 
be seen, daytime and nighttime images presented selectively and changeably on the 
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screen in display device 36 can be cross-related instantly to comparable thermal imagery 
presented dedicatedly on the screen in display device 34 (See particularly Figs. 6 and 7 
for thermal/daytime comparative imagery, and Figs. 8 and 9 for nighttime- 
intensified/thermal comparative imagery). Alternate daytime and light-intensified 
5 nighttime views, as during twilight, can be presented for comparison in succession on the 
screen in device 36 (See particularly Fig. 11 and 12, respectively), as well as for 
respective comparisons with same-scene thermal imagery (see Fig. 10). 

With focus now retumed for a moment to Fig. lA, shown in dash-double-dot, 
fragmentary lines 56, 58 at the right side of this figure are portions of two additional 

10 controllers which are like controller 28. These additional controllers can be employed, in 
accordance with one modification of system 10, to offer places for user control that are 
distributed to different locations. While two such additional controllers are shown at 56, 
58, it should be understood that any number of additional controllers, including only a 
single additional controller, may be employed advantageously if desired. 

15 Still considering systemic modifications that can be made, yet another 

modification is illustrated generally in Fig. IB. Here, in very simplified form, a 
controller 28 is shown operatively connected to a wireless transmitting device 58 which 
is designed to transmit control information fi-om controller 28 to operable equipment 
associated with imager housing 12, including all of the imagers provided therein, and the 

20 pitch and yaw drive motors. Information transmitted by device 58 is received by an 
appropriate receiver which is shown at 60 in Fig. IB, which receiver is suitably 
operatively connected to all of the controllable apparatus associated with housing 12. 
The wireless transmission medium employed may be a radio system, a wireless telephone 
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system, the Internet, and so on. A bracket 62 provided in Fig. IB is presented to 
emphasize the operative connectedness which exists between blocks 58, 60 in Fig. IB. 

Describing a bit more now about the use, in system 10, of controller 28, touch 
screen 28a and joystick 28b, in relation to establishing comparative views that may be 
5 presented in different ways on the screens in display devices 34, 36, let us turn attention 
to Figs. 3-5, inclusive. These three figures illustrate typical virtual control interfaces that 
may be presented on touch screen 28a to enable a system user, in conjunction with 
employment of joystick 28b, to implement full internal control over the operating 
parameters associated with imagers 14, 16 and 18, over the points-of-view (the aim) 

10 which may be selected, and over the selection of what modes of imagery to present on the 
screens in display devices 34, 36. Fig. 3 specifically illustrates a situation wherein the 
daytime (daylight) and the thermal imagers, 18, 16, respectively, are actively being used 
in the system. With these two imagers activated, comparative, like-scene imagery, such 
as that presented in Figs. 6 (thermal) and 9 (daytime), may be presented on the screens in 

15 display devices 34, 36, respectively. Thermal imagery is dedicatedly presented by device 
34, and in this illustration daytime, color imagery is selectively presented by display 
device 36 as a consequence of appropriate user selection of the operating condition for 
switching structure 32. 

Fig. 4 illustrates another typical virtual user interface presentation of virtual 

20 controls provided on touch screen 28a under circumstances where the nighttime and 
thermal imagers, 14, 16, respectively, are active. Under these circumstances, the 
user/operator of system 10 will have set the condition of switching structure 32 to send 
nighttime, light-intensified imagery to the screen in device 36. Device 34, of course. 
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presents thermal imagery. Figs 8 and 9 illustrate this comparative imagery condition in 
system 10, with Fig. 8 showing a display on the screen in device 36 of nighttime, light- 
intensified imagery, and Fig. 9 showing a comparable-scene thermal image on the screen 
in device 34. 

5 Considering the two touch-screen appearances which are shown in Figs. 3 and 4, 

and viewing these along with Figs 10, 1 1 and 12, one can think of these two touch-screen 
presentations as ones that might exist alternatively under user control on the touch screen 
under circimistances, say, at twiUght when the user/operator of system 10 decides that he 
or she wishes to look comparatively, along with (or without) thermal imagery (Fig. 10), 

10 at alternate, successive, nighttime, light-intensified imagery (Fig. 12) and daytime color 
imagery (Fig. 11) derived from the two associated imagers. Such nighttime and daytime 
imagery will be presented solely on the screen in display device 36 through operation, via 
touch screen 28a, of the operating condition of switching structure 32. As has just been 
suggested, Figs. 10 and 11 represent what might appear in time succession alternatively 

15 on the screen in device 36 under such circumstances. 

Fig. 5 in the drawings illustrates a typical control touch-screen interface 
particularly provided for the thermal imager. Such a screen might, for example, be 
employed as a precursor to preparing for the kind of "three-way", "two-screen" 
presentation pictured in Figs. 10, 1 1 and 12. 

20 Thus, a novel system and methodology for presenting useful, comparative 

surveillance imagery, derived from plural, different-mode imagers, are provided by this 
invention. Side-by-side display devices, fed information quickly, conveniently and 
selectively variably by the simple and intuitive one-hand-operable control structure 
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furnished in the system, offer a powerful co-viewing tool for gathering and accessing 
visual surveillance information. 

Accordingly, while a preferred embodiment (and certain modifications) of, and 
manner of practicing, the present invention have been described herein, it is appreciated 
5 that variations and modifications may be made without departing from the sprit of the 
invention. 
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